Method of providing a sharpness measure for an image

ABSTRACT

A method of providing a sharpness measure for an image comprises detecting an object region within an image; obtaining meta-data for the image; and scaling the chosen object region to a fixed size. A gradient map is calculated for the scaled object region and compared against a threshold determined for the image to provide a filtered gradient map of values exceeding the threshold. The threshold for the image is a function of at least: a contrast level for the detected object region, a distance to the subject and an ISO/gain used for image acquisition. A sharpness measure for the object region is determined as a function of the filtered gradient map values, the sharpness measure being proportional to the filtered gradient map values.

FIELD

The present invention relates to a method of providing a sharpnessmeasure for an image.

BACKGROUND

A. Santos, “Evaluation of autofocus functions in molecular cytogeneticanalysis”, Journal of Microscopy, Vol 188, Pt 3, December 1997, pp264-272 assesses a number of known sharpness measures. These can beclassified into five main groups as follows:

-   -   A. Functions based on image differentiation such as:        -   1. Threshold absolute gradient

${Sh}_{{th} - {grad}} = {\sum\limits_{M}{\sum\limits_{N}{{{g\left( {i,{j + 1}} \right)} - {g\left( {i,j} \right)}}}}}$

-   -   -   -   while |g(i,j+1)−g(i,j)|                -   >thr, where g(i,j) is the gray level of pixel (i,j)

        -   2. Tenengrad function

${Sh}_{tenengrad} = {\sum\limits_{M}{\sum\limits_{N}{T\left\lbrack {g\left( {i,j} \right)} \right\rbrack}}}$

-   -   -   -   where T[g(i,j)] is the square of the gradient value in                pixels (i,j)

    -   B. Functions based on depth of peaks and valleys

    -   C. Functions based on image contrast

    -   D. Functions based on histogram

    -   E. Functions based on correlation measurements including:        -   Vollath's F4 (based on the autocorrelation function, very            good performance in presence of noise)

${Sh}_{{VollathF}\; 4} = {{\underset{i = 1}{\sum\limits^{M - 1}}{\underset{j = 1}{\sum\limits^{N}}\left( {{g\left( {i,j} \right)} \cdot {g\left( {{i + 1},j} \right)}} \right)}} - {\underset{i = 1}{\sum\limits^{M - 2}}{\underset{J = 1}{\sum\limits^{N}}\left( {{g\left( {i,j} \right)} \cdot {g\left( {{i + 2},j} \right)}} \right.}}}$

All these functions perform pixel level computations providing aninstant sharpness value for a given image or a region of interest (ROI)within an image. In order to determine a best focus position for animage or a region of interest (ROI) within an image, a focus sweep mustbe executed so that the focus position indicating the highest sharpnesscan be chosen for acquiring an image. Performing such a focus sweepincluding assessing each image to determine an optimal focus positioncan involve a significant delay which is not acceptable, especially inimage acquisition devices where the ability to acquire a snap-shot or totrack an object in real-time is important.

None of these techniques is able to provide an absolute sharpness valuecapable of indicating if a region of interest is in focus when only asingle image is available, so indicating whether a change in focusposition might be beneficial in order to acquire a better image of ascene.

There are also other shortcomings of at least some of the aboveapproaches. Referring to FIG. 1, the top row shows a sequence of imagesof a face captured with the same face at different distances from acamera ranging from 0.33 m to 2 m. The light level for capturing theimages is similar ranging from 3.5 Lux to 2.5 Lux.

Referring to the respective image/graph pairs below the top row ofimages, the face region from each acquired image from the top row isscaled to a common size and in this case the upper half of the faceregion is chosen and scaled to provide a 200×100 pixel region ofinterest. For each of the scenes from the top row, the focus position ofthe camera lens is shifted by varying a code (DAC) for the lens actuatoracross its range from values, in this case from 1 to >61 and a sharpnessmeasure is calculated for each position. (Use of such DAC codes isexplained in PCT Application No. PCT/EP2015/061919 (Ref: FN-396-PCT) thedisclosure of which is incorporated herein by reference.) In thisexample, a threshold absolute gradient contrast measure such asdescribed above is used. Contrary to human perception, the sharpnessmeasure across the range of focus positions provided for the mostdistant 2 m image is actually higher than for the largest well-lit faceregion acquired at 0.33 m. This is because the sharpness measures forthe most distant image has been affected by noise.

Referring to FIG. 2, it will also be seen that in some of the abovecases, the sharpness measures for an image taken across a range of focuspositions, both on focused and on defocused images, in good light (20Lux) can be smaller than for those taken in low light (2.5 Lux),contrary to human perception, again because of the influence of noisewithin the image.

A different approach, which doesn't use a reference and provides aquality measure based on an eye band region is described in“No-Reference Image Quality Assessment for Facial Images” Debalina etal, pages 594-601, ICIC'11 Proceedings of the 7th InternationalConference on Advanced Intelligent Computing Theories and Applications.Debelina does not consider the behavior of this quality measure atvarious distances to a subject or in different lightning conditions. Themethod complexity is quite high involving k-mean clustering to separatethe eyes from the skin, binary template matching based oncross-correlation to detect the eyes, Just Noticeable Blur (JNB)thresholds to compute the sharpness and this may not be suitable for ahardware implementation.

It is an object of the present invention to provide a sharpness metricwhich reflects human perception of the quality of a ROI within an image.The metric should be valid for varying light levels including very lowlight conditions where an acquired image may be quite noisy. The metricshould be absolute so that a determination can be made directly from anygiven image whether it is sufficiently focussed or not i.e. thesharpness value for any well focused image should be higher than thesharpness level for any defocused image irrespective of an ambientluminance value.

SUMMARY

According to the invention there is provided a method of providing asharpness measure for an image according to claim 1.

It will be noted that the sharpness measure decreases with the distanceto the subject and with the ambient light level value in accordance withhuman perception.

The ideal absolute sharpness function should be narrow enough in allcases i.e. it should have a distinct peak around the peak focusposition.

Embodiments of the invention can be used to track a ROI over a number offrames, maintaining correct focusing, even though the ROI may movewithin the image and be exposed with various lighting conditions.

BRIEF DESCRIPTION OF THE DRAWINGS

An embodiment of the invention will now be described, by way of example,with reference to the accompanying drawings, in which:

FIG. 1 shows a number of images including a face region acquired atdifferent distances from a camera along with conventional type sharpnessmeasures for each image;

FIG. 2 shows a number of images acquired at the same distance but atdifferent light levels along with conventional type sharpness measuresfor each image;

FIG. 3 is a flow diagram illustrating the calculation of a sharpnessmeasure for a region of interest (ROI) of an image according to anembodiment of the present invention;

FIG. 4 shows the processing of FIG. 3 for an exemplar ROI of an image;

FIG. 5 shows a number of face regions from images acquired at differentdistances from a camera along with sharpness measures for each imageprovided according to an embodiment of the present invention; and

FIG. 6 shows a number of face regions from images acquired at differentlight levels along with sharpness measures for each image across a rangeof focus positions provided according to an embodiment of the presentinvention.

DESCRIPTION OF THE EMBODIMENTS

Referring now to FIG. 3, an embodiment of the present invention will nowbe described with reference to a ROI of interest of an image comprisingan object comprising a face.

The method begins by first identifying a region of interest (ROI) withinan image, step 10. In the present example the ROI bounds a face. It iswell known to be able to identify an object such as a face within animage using a variety of techniques based on classifiers and/or neuralnetworks such as disclosed in PCT Application No. PCT/EP2016/063446(Ref: FN-471-PCT). In the case of a face, once identified within theimage, anthrometric information can be used to assist with the methodsuch as knowing that the distance between eyes in an adult isapproximately 7 cm as disclosed in PCT/EP2015/076881 (Ref: FN-399-PCT)or that the smallest size of face which might need to be focus trackedwould measure approximately 200×200 pixels.

Nonetheless, it will be appreciated that the present invention isequally applicable to providing a sharpness measure for an image with aROI containing any type of object with a view to trying to provide anabsolute sharpness value for any given image reflecting humanperception.

At step 12, the method acquires the metadata for the currentobject/frame. This includes, for example, ISO/Gain, exposure time,object ROI coordinates, object pixel values. Other meta-data which canbe used within embodiments of the invention could involve indicators ofobject orientation. It will be appreciated that the characteristicsexhibited by an object such as a face differ according to itsorientation within an image, for example, a profile image will onlyinclude 1 eye and might contain proportionally more skin than a frontfacing face. Similarly faces which are identified under differentillumination conditions (for example, using differently trainedclassifiers such as: top-lit, side-lit etc.) can exhibit differentcharacteristics. This information can be used in variations of theembodiment disclosed below, for example, to vary parameters employed inthe sharpness measure. On the other hand, some parameters can be set totake into account the characteristics of features such as the eyes,skin, face pattern, face geometry/dimensions implicitly as will beexplained below.

At step 14, the upper part of the face region i.e. the portioncontaining the eyes is chosen as the ROI on which the sharpness measurewill be calculated. However, it will be appreciated that the measurecould be based on the complete face region. Also, it will be clear thatfor inverted or portrait images, the upper part of the face may appearbelow or to the left/right of the lower part of the face—this is readilydealt with to ensure the portion containing the eyes is chosen.

In the embodiment, the face size for sharpness computation is chosen as200 (width)×100 (height) as, typically, this is the smallest face sizewhich might be required for AF/sharpness evaluation. Thus, in step 15,the selected upper part of the face region is scaled to a size of200×100 to provide a ROI 40 such as shown in FIG. 4.

At step 16, a series of thresholds are calculated as follows:

ISO Threshold (thr_ISO)=Meta Data ISO Value/250;

Luminance Threshold (thr_Lum)=Average luminance for pixels of theROI/50;

Distance to subject threshold (thr_Dist)=ROI width/200;

Sharpness Threshold=max(12,thr_ISO+thr_Lum+thr_Dist).

The constants used in the ISO, Luminance and Distance thresholdcalculations above normalize each component of the sharpness thresholdrelative to one another and are based on the face size chosen forsharpness computation. As such, these can vary if a region of interestwith a size different than 200×100 were chosen. Thus for a larger ROI,these constants would increase (so reducing the sharpness measure aswill be seen later).

It will also be appreciated that while it is expected that an imageincluding higher luminance values than an image with lower luminancevalues would be of higher quality and so its contribution to thesharpness threshold would be opposite that of ISO/gain (where increasinglevels indicate poorer lighting), the present embodiment uses theaverage luminance value as a quick measure of the likely contrast rangewithin an image—contrast tending to increase with increasing luminancevalues and so indicating higher noise levels within an image. Thus, invariants of the embodiments other measures of contrast level within theROI could be employed than using average luminance. In this regard, itwill be noted that using the ROI width as a measure of the distance tothe subject involves minimal calculation and so speeds up this method.

The above sharpness threshold also assumes that optimal focus positionis being determined using preview images, prior to capturing a mainimage. Typically, exposure time for such images is maximal and so wouldnot vary. On the other hand, if exposure time were to be less thanmaximal i.e. variable, then the sharpness threshold would beproportional to the exposure time component.

At step 18, a raw sharpness (gradient) map is provided by calculating asimple difference between the luminance channel (Y) values for thescaled 200×100 ROI and a version of the scaled 200×100 ROI shifted by 1pixel to the right. It will be seen that many different techniques canbe used to provide a gradient map, for example, using a histogram ofgradients (HOG) map for an image and shifting the ROI images differentlyrelative to one another.

Now the resulting raw sharpness map is filtered by a simple thresholdingwhere the sharpness threshold calculated at step 16 above is firstsubtracted from the gradient map values, step 20. Referring to FIG. 4,for a scaled ROI 40, the filtered sharpness map might look like the map42.

This method now takes advantage of the fact that an important percentageof the ROI contains relatively uniform skin regions, where the sharpnessis expected to be low.

Thus in step 22, the filtered gradient map is split into an array ofgenerally equal sized cells. In the example of FIG. 4, an array of 7×5cells is employed.

The mean sharpness of each cell is calculated as the average of thefiltered gradient map values in each cell 24. Sample values for the ROI,map and array 40-44 are shown in the array 46 of FIG. 4.

These values are next sorted in order of magnitude, step 26, as shown inthe list 48.

At step 28, the values 50 for index [3] to index [13] within the orderedlist 48 are selected and their average is chosen as an indicator of thenoise level within the ROI 40. Thus, in the present example, where theskin pixels are smooth rather than noisy, a noise level of 0 isdetermined for the ROI 40.

At step 30, a raw sharpness measure is calculated as the mean value ofthe filtered gradient map produced in step 20 less the noise leveldetermined at step 28. Now the raw sharpness measure can be normalizedwith respect to luminance by dividing the raw sharpness measure by themean luminance for the ROI to provide a final sharpness value, step 32.

It has been found that the above method is robust to face rotation andpartial face obtrusion (by hair, glasses) particularly as the noiselevel computation for the ROI is determined based on a reduced sortedvector.

FIG. 5 illustrates the sharpness measure calculated according to theabove embodiment for the faces shown in FIG. 1 i.e. the same face, takenin almost same ambient conditions where only the distance to the subjectis varying. As can be seen, at 2 m the absolute sharpness values aresmaller than at 0.33 m, similar to the human perception.

FIG. 6 illustrates the sharpness measure calculated according to theabove embodiment for the faces shown in FIG. 2 i.e. on faces at samedistance at various ambient light levels. As will be seen, for goodlight, the sharpness measure is bigger than for images acquired in inlow light, again similar to the human perception, and with awell-defined peak.

The sharpness value for the focused images is higher than the sharpnessvalues of the defocused images even when we compare the curves taken indifferent lighting conditions. This is again similar to the humanperception.

A second embodiment can improve the quality (from a human perceptionpoint of view) of the sharpness measure of the above described firstembodiment for images of subjects at shorter distances (e.g. 300 mm-500mm). Compared to images of subjects at longer distances (>1000 mm), alarger quantity of light lands on the image sensor for images ofsubjects at shorter distances. Thus, a wider range of lens positionstends to output similar sharpness levels, when the lens is near adesired shorter focus position.

However, with more information in such images, stronger filtering can bedone and thus the focus position can be determined for such images moreaccurately i.e. the sharpness curve slope should increase and provide aclearer peak value.

In the previous embodiment, the image is split in 7×5 blocks 44 and theaverage illumination for each block is stored in an array 46. The arrayvalues are then sorted in ascending order and a noise level is extractedat step 28, using the 3^(rd) and 13^(th) elements of the sorted array(sorted_vals) 48.

Instead of using just these elements, the second embodiment takes intoaccount all of the elements from the sorted array 48 and splits theminto two categories:

-   -   First 30 elements, from 1^(st) to 30^(th) (lowest        values)—comprise residual information with a higher probability        of containing noise and a lower probability of containing useful        information (e.g. eyes)    -   Last 6 elements, from 30^(th) to 35^(th) (highest        values)—comprise useful information with a higher probability of        containing information from the eye and eyebrow regions and with        a lower probability for noise.

Using the same thresholds (thr_Dist, average luminance) as in the firstembodiment, together with the ISO value, two weights corresponding tothe useful information (w_(b)) and the residual information (w_(r)) arecomputed as follows:

w _(b)=0.1*(thr_Dist−(ISO Value/Average luminance)*0.01);

w _(r)=1−w _(b);

Thus, extracting the noise level at step 28 of the first embodiment isreplaced with extracting a face threshold parameter (thr_Face) byassigning the above mentioned weights in the form of a weighted averageas follows:

thr_Face=mean(sorted_vals(30:35))*w _(b)+mean(sorted_vals(1:30))*w _(r);

with the initial sharpness threshold of the first embodiment beingadjusted as follows:

sharpness_thr=min([MAX_SHARPNESS_TH,(max(12,thr_ISO+thr_Lum+thr_Dist)+min(10,thr_Face*20))]);

where MAX_SHARPNESS_TH is a constant=20.

Numerically, this allows for a stronger filtering at small distances,lowering the sharpness values for the blurred pictures and generating abetter determined peak value.

In step 30 of the first embodiment, a raw sharpness value is computed bysubtracting the noise level from the mean of the filtered gradient mapproduced in step 20 in order to filter the residual information from theimage, for example, as follows:

raw_sharpness=max(0,(mean filtered gradient map−noise_level)*1000)

In contrast, in the second embodiment, thr_Face replaces the noise levelin the above formula, for example, as follows:

raw_sharpness=max(0,(mean filtered gradient map+thr_Face)*1000).

The reason is that the face threshold quantifies the useful informationand provides for improved sharpness discrimination at differentdistances. Note that for each of the above steps further constants maybe employed within the calculations described in order to scale theintermediate values and indeed the final sharpness value as required.

Indeed these and the other values described above can be varied from oneimage device to another to take into account different scales of valuesused across different devices.

While the above described embodiments produce a useful absolutesharpness value across a range of image acquisition conditions, it willbe seen that there can come a point where the image is so noisy that themeasure may not be reliable. Thus further variants of the abovedescribed embodiments can take this into account and flag that providinga measure is not possible or that the measure is not reliable.

1. A method of providing a sharpness measure for an image comprising thesteps of: detecting an object region within an image; obtainingmeta-data for the image; scaling the chosen object region to a fixedsize; calculating a gradient map for the scaled object region; comparingthe gradient map against a threshold determined for the image to providea filtered gradient map of values exceeding the threshold; determining asharpness measure for the object region as a function of the filteredgradient map values, the sharpness measure being proportional to thefiltered gradient map values; wherein the threshold for the image is afunction of at least: a contrast level for the detected object region, adistance to the subject and an ISO value used for image acquisition. 2.A method according to claim 1 wherein the method further comprisesscaling the chosen object region to a fixed size.
 3. A method accordingto claim 2 wherein the fixed size is 200×100 pixels.
 4. A methodaccording to claim 1 wherein the object is a face.
 5. A method accordingto claim 4 wherein the method further comprises cropping the face regionto only use a face region containing at least one eye as the chosenobject region.
 6. A method according to claim 1 comprising calculating anoise measure for the object region by: splitting the filtered gradientmap into an array; determining a mean sharpness for each cell of thearray; sorting the mean sharpness values in order of magnitude;selecting a sub-range of sharpness values from the sorted sharpnessvalues; and calculating said noise measure for the object region as afunction of said sub-range of sharpness values.
 7. A method according toclaim 6 wherein said sub-range comprises a group of the lowest valuedsharpness values.
 8. A method according to claim 6 wherein the sharpnessmeasure is a function of the noise measure for the object region, thesharpness measure being inversely proportional to the noise measure. 9.A method according to claim 1 comprising calculating a threshold measurefor the object region by: splitting the filtered gradient map into anarray; determining a mean sharpness for each cell of the array; sortingthe mean sharpness values in order of magnitude; the threshold for theobject region, thr_Face, being defined as follows:thr_Face=mean(sorted_vals(high))*w _(b)+mean(sorted_vals(low))*w _(r);where: sorted_vals(low) and sorted_vals(high) are groups of lowest andhighest mean sharpness values respectively;w _(b) αf(distance to subject)−(ISO value/contrast level*d);w _(r)=1−w _(b); and d is a constant.
 10. A method according to claim 9wherein the sharpness measure is proportional to the threshold measurefor the object region.
 11. A method according to claim 9 wherein thethreshold for the image is proportional to sum of thresholds for ISO,contrast and distance adjusted by said threshold for the object region,thr_Face.
 12. A method according to claim 1 wherein the threshold forthe image is the maximum of: a constant; and a sum of thresholds forISO, contrast and distance.
 13. A method according to claim 1 whereinthe threshold for the image is proportional to average luminance, ISOand distance.
 14. A method according to claim 1 further comprisingextracting meta-data from an acquired image to determine parameters foreach of said ISO, contrast and distance thresholds.
 15. An imageprocessing device arranged to perform the method of claim
 1. 16. Acomputer program product comprising computer readable instructions,which when executed in an image processing device are arranged toperform the method of claim 1.